• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    A method implemented by computer to simulate the behavior of a hand that interacts with objects in a virtual environment, which comprises acquiring a skeleton (202) of a hand (300) using a hand tracking device (620); computing a hand (310) simulated with objects (222) in a virtual environment (220); establish a connection between the skeleton (214) of a simulated hand (310) and the skeleton (202) followed using connection energies (Ucrb); establish a simulated internal hand power (310) that takes into account skeletal joint constraints, skeleton-tissue connection, and nonlinear soft tissue deformation; establishing a friction contact between the simulated hand (310) and the virtual objects using a penalty energy considering the relative interpenetration at contact points and a friction energy considering the deviation between each contact point and its corresponding sliding anchor point; compute an updated skeleton (214) and a simulated hand tissue (216) (310) minimizing the total energy involved. (Machine-translation by Google Translate, not legally binding)
    公开号:ES2725128A1
    申请号:ES201830260
    申请日:2018-03-15
    公开日:2019-09-19
    发明作者:Mickeal Verschoor；Tristran Miguel Angel Otaduy；Cuenca Daniel Lobo
    申请人:Universidad Rey Juan Carlos；
    IPC主号:
    专利说明:

    [0001] METHOD IMPLEMENTED BY COMPUTER, SYSTEM AND PRODUCT OF THE PROGRAM
    [0002]
    [0003]
    [0004]
    [0005] Field of the Invention
    [0006]
    [0007] The present invention is comprised within the field of real-time simulations of the behavior of a human hand that interacts with virtual objects.
    [0008]
    [0009] Background of the invention
    [0010]
    [0011] Hands are our main means to interact with the world. Due to its importance, it is not surprising that the scientific community there witnessed a multitude of methods for monitoring, simulation, animation, robotics or hand-based touch systems. All these areas could benefit from a ready-to-use hand-held simulation model, which is also deficient, which can provide a robust and smooth interaction.
    [0012] Over the years the simulation of humans and human skin has achieved a development towards realism. The first models of human hands were based solely on rigid bodies and provided some form of force feedback to the user and a connection was made between an articulated and followed hand ([2]; [19]; [11]). Although the grip of the object is possible, due to the absence of a deformable skin, an accurate simulation of friction restrictions was not possible.
    [0013]
    [0014] Simulating a human hand that includes a deformable skin requires simulating all the different aspects that ultimately form the human hand. This includes the skeleton, muscles, tendons and skin. In these simulations, it is important that each different aspect is properly connected with the other parts so that realistic behavior is obtained, which makes the problem difficult to solve. Depending on the level of detail required, there are several approaches. Highly detailed approaches are for example found in [[28], where the hand is simulated with a very high level of detail, which includes for example the muscles. Other approaches perform deformation in a geometric manner, but deformations due to physical interactions are often not modeled ([12];
    [0015] [13]). Other methods use data-based approaches to model hand deformation by learning from examples. Since these methods are not completely based on the physical, it is difficult to obtain a good deformation while the hand is in contact with an object ([26]; [16]; [14]).
    [0016]
    [0017] Using detailed hand simulations with deformable skin in interactive applications and in real time requires a balance between realism and computer costs. To reduce the computational complexity of simulating deformable hands, often only the fingers are simulated. This greatly reduces the complexity of the model used ([1]; [6]; [3]; [31]). Realistic hand simulations incorporate a rigid skeleton simulation with a deformable skin connected to the bone structure. Methods that connect a portion of the deformable skin to the skeleton are for example ([10]; [27]; [5]). More detailed approaches also simulate parts of skin deformation, as in [22].
    [0018]
    [0019] An additional stage incorporates the simulation of the complete skin that is connected to the bones in a bidirectional connection. This means that the forces in the skin will spread to the bones, and that the bones can generate, as a result, forces in the skin. Such an approach greatly increases the realism of deformable hands. The approach of [7] makes the connection between the bones and the skin in two stages. First the approximate configuration of the skeleton is computerized which also takes into account joint restrictions. Then the skin configuration is calculated given the configuration of the bones, and the bones are allowed to update the configuration of the skeleton. The contact is also modeled using strong restrictions. In addition, the authors introduce a modified version of a virtual connection [4]. Pérez et al. ([20]; [25]) investigates the simulation of non-linear skin based on limited tension restrictions and presents an efficient method to solve these types of restricted problems [21].
    [0020]
    [0021] Other related works can be found in [8], in which a detailed high-resolution skin model is simulated, which is formed by the underlying skeletal configuration. In this work, there is no additional simulation of the skeleton, nor a virtual connection between the hand configuration followed and the skin simulation. This results in too large tensions and deformations in the skin since the simulation does not restrict the movement of the user's hand, or compensates for it. Work on [15] skin shape using a global skin blend based on the underlying skeletal configuration. In this case, a physics-based simulation for the skeleton and skin is not performed. However, above all this, local deformation is computed based on the depth of penetration of the skin and on a seized object.
    [0022] It is therefore necessary to find a method implemented by computer that performs a real-time simulation of a human hand that improves the response, realism and stability of the simulation, and at the same time that requires a minimum computing cost.
    [0023]
    [0024] References
    [0025]
    [0026] [1] F. Barbagli, A. Frisoli, K. Salisbury, and M. Bergamasco. Simulation of human fingers: a model and algorithm representative of soft fingers. In HAPTICS ‘04. Procedures 12th International Symposium in 2004
    [0027]
    [0028] [2]. C. W. Borst and A. P. Indugula. Realistic virtual grip. In Proc. of IEEE, Virtual Reality Conference, 2005.
    [0029]
    [0030] [3] M. Ciocarlie, C. Lackner, and P. Allen. Soft finger model with adaptive contact geometry for grip and handling tasks. At World Haptic Conference, 2007.
    [0031]
    [0032] [4] J. E. Colgate, M. C. Stanley, and J. M. Brown. Problems in the haptic visualization of the use of tools. Proc. of IEEE / RSJ, International Conference of Robots and Intelligent Systems, pages pp. 140-145, 1995.
    [0033]
    [0034] [5] Christian Duriez, Hadrien Courtecuisse, Juan-Pablo de la Plata Mayor, and Pierre- Jean Bensoussan. Skin contact Eurographics Conference (short document), 2008.
    [0035]
    [0036] [6] A. Frisoli, F. Barbagli, E. Ruffaldi, K. Salisbury, and M. Bergamasco. A boundary curve based on an all-powerful soft finger algorithm. In Haptic Interfaces for Virtual Environment and Telemarketer Systems, 14th Symposium in 2006.
    [0037]
    [0038] [7] Carlos Garre, Fernando Hernandez, Antonio Gracia, and Miguel A. Otaduy. Interactive simulation of a deformable hand for haptic representation. In Proc. of the World Haptic Conference, 2011.
    [0039]
    [0040] [8] Koichi Hirota and Kazuyoshi Tagaw. Interaction with a virtual object using a deformable hand. 2016 IEEE, Virtual Reality (VR), 00: 49-56, 2016.
    [0041]
    [0042] [9] G. Irving, J. Teran, and R. Fedkiw. Invertible finite elements for robust simulation of great deformation. Proc. from ACM SIGGRAPH / Eurographics Symposium in computer animation, pages 131-140, 2004.
    [0043] [10] J. Jacobs and B. Froehlich. A soft hand model for a physically based manipulation of virtual objects. In 2011 IEEE Virtual Reality Conference (VR), pages 11-18, March 2011.
    [0044]
    [0045] [11] J. Jacobs, M. Stengel, and B. Froehlich. A generalized almighty object method for plausible finger-based interactions in virtual environments. In 2012 IEEE, Symposium on 3D User Interfaces (3DUI), pages 43-51, March 2012.
    [0046]
    [0047] [12] Paul G. Kry, Doug L. James, and Dinesh K. Pai. Figenskin Great deformation in real time of skin characterization in hardware. In ACM SIGGRAPH / Symposium Eurographics in computer animation, pages 153-160, July 2002.
    [0048]
    [0049] [13] Tsuneya Kurihara and Natsuki Miyata. Modeling of deformable human hands of medical images. In 2004, ACM SIGGRAPH / Symposium Eurographics in Computer Animation, pages 355-363, July 2004.
    [0050]
    [0051] [14] Paul G. Kry and Dinesh K. Pai. Capture interaction and synthesis. ACM transactions in graphs 25 (3): 872-880, July 2006.
    [0052]
    [0053] [15] Jun-Sik Kim and Jung-Min Park. Physically based hand interaction with virtual objects. At IEEE, International Robotics and Automation Conference, ICRA 2015, Seattle, WA, USA, May 26-30, 2015, pages 3814-3819, 2015.
    [0054]
    [0055] [16] Ying Li, Jiaxin L. Fu, and Nancy S. Pollard. Data-based grip synthesis using a matching form and task-based file deletion. IEEE, Transactions in Visualization and Computer Graphics, 13 (4): 732-747, July / August 2007.
    [0056]
    [0057] [17] D. Lobo, M. Sarasc, M. Verschoor, M. Solazzi, A. Frisoli, and M. A. Otaduy. Proxy-based haptic representation for underutilized haptic devices. In 2017 IEEE, World Haptic Conference (WHC) IEEE, pages 48-53, June 2017.
    [0058]
    [0059] [18] Matthias Müller and M. Gross. Interactive virtual materials Proc. of Graphic Interface, 2004.
    [0060]
    [0061] [19] R. Ott, F. Vexo, and D. Thalmann. Two-handed haptic manipulation for DAO and RV applications. Computer Aided Design and Applications 7 (1), 2010.
    [0062]
    [0063] [20] Alvaro G. Perez, Gabriel Cirio, Fernando Hernandez, Carlos Garre, and Miguel A Otaduy. Voltage limitation for simulation of soft finger contact. In Proc. of IEEE, World Haptic Conference, 2013.
    [0064] [21] Alvaro G. Perez, Gabriel Cirio, Daniel Lobo, Francesco Chinello, Domenico Prattichizzo, and Miguel A. Otaduy. Efficient non-linear skin simulation for multi-finger touch representation. In symposium of procedures of Haptic. IEEE, 2016.
    [0065]
    [0066] [22] M. Pouliquen, Christian Duriez, C. Andriot, A Bernard, L. Chodorge, and Florian Gosselin. Simulation of finger pinch grip by finite elements in real time. At Eurohaptics Conference, 2005 and Symposium on Haptic Interfaces for Virtual Environment and Telemarketer Systems, 2005. World Haptics 2005. First meeting, pages 323-328, March 2005.
    [0067]
    [0068] [23] Theodore Papadopoulo and Manolis I. A. Lourakis. Jacobian estimation of the singular value decomposition: theory and applications. In European Conference on Computer Vision, 2000.
    [0069]
    [0070] [24] A. G. Perez, D. Lobo, F. Chinello, G. Cirio, M. Malvezzi, J. S. Martín, D. Prattichizzo, and M. A. Otaduy. Portable touch representation based on optimization. IEEE, Haptic Transitions, 10 (2): 254-264, April 2017.
    [0071]
    [0072] [25] Jesus Perez, Alvaro G. Perez, and Miguel A. Otaduy. Simulation of hyper-elastic materials using energy restrictions. In Proc. from CEIG, 2013.
    [0073]
    [0074] [26] Nancy S. Pollard and Victor B. Zordan. Grip control physically based on an example. In 2005, ACM SIGGRAPH / Symposium Eurographics of Computer Animation, pages 311 318, July 2005.
    [0075]
    [0076] [27] Alec R. Rivers and Doug L. James. FastLSM: fast lattice shape matching for robust deformation in real time. ACM Transactions in Graphics (procedures of SIGGRAPH 2007), 2007.
    [0077]
    [0078] [28] Shinjiro Sueda, Andrew Kaufman, and Dinesh K. Pai. Muscle-tendon simulation for hand animation. ACM Trans. Graph., 27 (3), August 2008.
    [0079]
    [0080] [29] F. Sin, Y. Zhu, Y. Li, D. Schroeder, and J. Barbic. Isotopic hyper-reality invertible using SVD gradients. In ACM SIGGRAPH / Symposium Eurographics in Computer Animation (posters), 2011.
    [0081]
    [0082] [30] J. Teran, S. Blemker, V. Ng Thow Hing, and R. Fedkiw. Finite volume methods for simulation of a skeletal muscle. In SIG-GRAPH / Eurographics Symposium in Computer Animation, pages 68-74, 2003.
    [0083] [31] C.B. Zilles and J.K. Salisbury An all-powerful object method based on restriction for haptic visualization. In Robots and Intelligent Systems, International Conference, IEEE / RSJ in, volume 3, page 3146. Los Alamitos, CA, USA, 1995. IEEE Information Society.
    [0084]
    [0085] Description of the invention
    [0086]
    [0087] The present invention relates to a method implemented by computer with an efficient soft hand simulation model, which achieves a natural hand-based interaction that presents a smooth hand movement that adapts to the user's movement followed, reacts Robust contact with objects in a virtual environment, and allows the right handling of these objects. The hand simulation model integrates an articulated skeleton, a non-linear soft tissue and a friction contact, to provide the realism necessary for natural interaction. The robust and smooth interaction is made possible by simulating in a single framework of minimization energy all mechanical exchanges between hand elements. Connection between the skeleton of the hand and the movement of the user, restrictions on the joints of the skeleton, non-linear soft skin formation, connection between the skeleton of the hands and soft skin, friction contact between the skin and virtual objects , and connection between seized objects and other virtual objects. All the elements of the hand that provide realism and natural integration are taken into account, while ensuring a minimum and limited computer cost, which is the key to a smooth and robust integration. As a result, the proposed method for hand simulation can be connected to various input tracking devices, and seamlessly integrated into game engines for rapid development in virtual reality (VR) applications.
    [0088]
    [0089] A first aspect of the invention relates to a method implemented by computer to simulate the behavior of a hand that interacts with objects in a virtual environment. The method comprises acquiring, repeatedly, a hand configuration of a hand followed from data captured by a hand tracking device, the hand configuration that includes information to establish the position and orientation of the bones that form the skeleton followed by the hand followed; and repeatedly computing a hand configuration of a simulated hand in a hand simulation that interacts with at least one object in a virtual environment, the simulated hand comprising a non-linear deformable tissue and an articulated skeleton with joints and bones.
    [0090]
    [0091] The method also comprises the following stages:
    [0092] - Establish a connection between the simulated hand skeleton and the hand skeleton followed using connection energies between corresponding bone pairs considering their relative positions and orientations.
    [0093]
    [0094] - Establish an internal energy of the simulated hand that takes into account skeletal joint restrictions, skeleton-tissue joints, and nonlinear soft tissue deformation. In particular, the method determines an internal energy of the simulated hand, the internal energy comprising a potential energy associated with the skeletal joints, a potential energy associated with the tissue, and an energy function associated with the union between the tissue and the tissue. skeleton. In particular, the method establishes a friction contact between the simulated hand and at least one object in the hand simulation using potential energies in the normal and tangential directions between a node of the simulated hand tissue and an anchor node on the surface of the at least one object in the hand simulation.
    [0095]
    [0096] - Establish a friction contact between the simulated hand and other objects using a penalty energy that considers the relative interpenetration at contact points and a friction energy that considers the deviation between each contact point and its corresponding sliding anchor point.
    [0097]
    [0098] - Repeatedly compute an updated skeleton and simulated hand tissue minimizing the total energy involved.
    [0099]
    [0100] The step of establishing a connection between the simulated hand skeleton and the skeleton followed can comprise the following sub-stages:
    [0101]
    [0102] Establish a connection between a synchronized copy of the simulated hand skeleton and the intermediate skeleton using binding energies between each pair of corresponding bones.
    [0103] Establish a connection between the skeleton followed and the intermediate skeleton using connection energies between each pair of corresponding bones.
    [0104]
    [0105] Establish a connection between a synchronized copy of the intermediate skeleton and the simulated hand skeleton using connection energies between each pair of corresponding bones.
    [0106] Repeatedly compute an updated intermediate skeleton minimizing the connection energies involved. This is particularly useful when the computation of an updated intermediate skeleton is performed at a faster rate than the computation of an updated simulated hand skeleton.
    [0107] According to one embodiment, the step of computing a simulated hand hand configuration comprises modeling the joints using potential energies to hold the endpoints of the bones to form the joint together and limit the angle of the joint.
    [0108]
    [0109] In one embodiment, the step of computing a simulated hand configuration comprises computing the deformation of the tissue using a finite element method, wherein the deformable tissue is described considering a potential energy having an energy density function. quadratic potential, and where the computer domain is divided into a set of tetrahedron elements.
    [0110]
    [0111] The step of computing a simulated hand hand configuration may comprise establishing a connection between the skeleton and the simulated hand tissue using an energy function between bones in the skeleton and nodes in the tissue, corresponding to the intersection of bones and tissues.
    [0112]
    [0113] The object in the virtual environment can be a rigid body or a flexible body. In the first case, the method comprises establishing a connection between the object that interacts with the simulated hand in the hand simulation and the corresponding simulated object in the virtual environment that uses connection energies considering the difference in position and orientation between pairs of objects, and repeatedly compute an updated object in hand simulation minimizing connection energies. When the object is a flexible body, the method comprises establishing a connection between the object that interacts with the simulated hand and the hand simulation and the corresponding object simulated in the virtual environment using connection energies considering the difference in the state between pairs of objects, and repeatedly compute an updated object in hand simulation minimizing connection energies.
    [0114] The method can also comprise representing on a display device the simulated hand interacting with it at least one object in the virtual environment.
    [0115]
    [0116] Another aspect of the present invention relates to a system to simulate the behavior of a hand that interacts with objects in a virtual environment. The system comprises a data processing device, such as a processor or a computer, configured to carry out the steps of the method implemented by computer.
    [0117]
    [0118] The system can also comprise any of the following additional elements:
    [0119] - the handheld tracking device for repeatedly acquiring the handheld hand configuration followed by a user.
    [0120] - A display device (for example a monitor or a head-mounted screen) configured to represent the simulated hand interacting with it at least one object in the virtual environment.
    [0121]
    [0122] - A portable handheld haptic device (for example, a glove) configured to apply a haptic feedback response on the hand followed by a user according to the forces computerized by the data processing device at simulated hand contact points , said forces being generated by integration with objects in the virtual environment.
    [0123]
    [0124] A third aspect of the present invention relates to a computer program product to simulate the behavior of a hand that interacts with objects in a virtual environment. The computer program product comprises a computer program code usable by computer to, when executed in a processor, perform the steps of the previously implemented computer implemented method.
    [0125]
    [0126] A fourth aspect of the present invention relates to a computer readable storage medium comprising instructions to cause a processor to carry out the steps of the previously defined computer implemented method.
    [0127]
    [0128] The method implemented by computer performs a real-time simulation of the skeleton, the skin, the contact and connection energies of a human hand as a large simulation, rather than solving each of the parts separately from the rest, therefore significantly improving simulation stability. Especially, when the user is grabbing virtual objects, a stable simulation of all facets of the virtual hand is of paramount importance. For a stable and realistic connection of user movement and simulation, the virtual connection procedure presented in [7] is followed. Instead of modeling joints using strong speed restrictions, energy based on weak restrictions is used, which significantly increases the response and stability of the solver. Similarly, contact and friction are modeled through penalty energies, both for contact and friction. Kinetic friction is computed by maintaining friction anchors. When the energy density in the skin exceeds a certain maximum, tension and deformation are limited using a penalty energy density such that the behavior of non-linear materials can be approximated. Since no strong restrictions are used, the convergence of the method remains predictable, which is required for real-time applications. In addition, hand simulation can be connected with other simulations and / or existing virtual environments.
    [0129] The present hand model can be used to interact with objects in virtual worlds through a mechanical exchange of forces and energies between virtual objects and the hand model. This approach differs significantly from existing solutions in which interactions with objects and / or with the environment are possible through some gestures. In these existing solutions, the user makes a hand gesture (such as closing the hand) and some tasks are performed with nearby objects. For example, grabbing an object by fixing the object in the hand by making the “closed hand” gesture. However, in the present method a virtual hand follows the movement of the user's hand obtained through some hand tracking method, which captures the position and configuration of the user's hand. If the user closes his hand and some virtual object is between the fingers of the virtual hand, the virtual hand and the virtual object will be in contact. The user is now able to interact with the virtual object when doing it with a real object By opening or closing the hand further, the user can change the amount of force applied to the virtual object This also changes the frictional forces between the virtual object and the finger.
    [0130]
    [0131] The virtual hand is modeled through conservative energy models for each different aspect of the virtual hand. The virtual hand is an articulated hand model that includes the bones and joints in the hand. The skin of the virtual hand is modeled using a soft non-linear fabric that is wrapped around the articulated skeleton. The contact between non-linear skin and other objects is modeled through penalty energies, which include friction energies. To connect the virtual hand movement and an example followed by the skeleton (obtained by the hand tracking method), conservative energies are used as a function of the position and angular deviation between the skeleton in the hand and the configuration followed . The connection to the virtual environment is made similarly using conservative energies as a function of the deviation between two examples of the same object seized. An example is simulated in the virtual world, the other is simulated in contact with the virtual hand. This allows stable interaction with a simulated object in the virtual environment, and a reuse of existing ready-to-use physical engines used within virtual environments. All parts of the simulation are simulated using an organization based on an implicit Euler method, which seeks an optimal solution that conserves total energy in the system.
    [0132]
    [0133] Brief description of the drawings
    [0134]
    [0135] A series of drawings that help a better understanding of the invention and that are expressly related to an embodiment of said invention, presented as a non-limiting example thereof, are briefly described below.
    [0136] Figures 1A and 1B represent two examples of a real-time hand-based interaction with an object, in accordance with the computer-implemented method of the present invention.
    [0137]
    [0138] Figures 2A and 2B schematically represent, in two different embodiments, the connection of all parts in the simulation.
    [0139]
    [0140] Figures 3A and 3B illustrate an overview of the complete model and a hand simulation when interacting with one or two rigid bodies, respectively.
    [0141]
    [0142] Figure 4A represents the configuration in the first impact between the skin of the virtual hand and an object. Figure 4B shows an update of the anchor node when a kinetic friction is applied.
    [0143]
    [0144] Figure 5 shows a schematic overview of hand simulation as a library for integration with third-party tools.
    [0145]
    [0146] Figure 6 represents an embodiment of a hardware configuration used to run a handheld simulation.
    [0147]
    [0148] Description of the embodiments of the present invention
    [0149]
    [0150] Figures 1A and 1B represent a natural and intuitive real-time hand-based interaction. Accurate right hand manipulation is possible thanks to the efficient and robust simulation of a soft hand model. The method implemented by computer performs a simulation of a deformable hand 102 in a virtual environment 220, which is controlled by a hand tracking solution, and is capable of interacting with virtual objects 222 that are simulated in the virtual environment 220 with a motor third party physicist
    [0151]
    [0152] Figure 2A represents a schematic overview of all the individual parts identified in the simulation and shows which parts will interact with each other through energy transfer. The black examples are synchronized copies of the corresponding entities (objects, skeleton) and act as static (non-mobile) entities in the simulation. When your blank equivalents are updated, the synchronized copy is also updated. The simulation included in the hand tracking, the tracking loop 200, is affected by the skeleton 202 followed by the hand followed (obtained from the data captured by the follower input) and a synchronized copy of the skeleton 204 of the simulation 210 of hand, while the intermediate skeleton 206 (a virtual intermediate layer used to connect the computerized hand configuration in the hand simulation 210 with the hand configuration captured by hand tracking device) is updated at high simulation speed. When a new simulation stage in the hand simulation 210 is initiated, the properties of a synchronized copy of the intermediate skeleton 212 in the hand simulation 210 are updated and will affect the skeleton 214, tissue 216 and the bodies or objects 218 that are in Contact. The connection to the virtual environment 220 (a third party simulation performed by a physical engine) is carried out in a similar way, except that speeds and accelerations (forces and moments) are also transferred. This is due to a possible lower simulation speed.
    [0153]
    [0154] The tracking loop 200 is preferably used when the tracking process to obtain the skeleton 202 followed is executed at a higher speed than the computation performed in the hand simulation 210 to have an updated simulated hand 310. In another embodiment, depicted in Figure 2B, in which the simulation speed for computing the simulated hand 310 is close to or equal to the speed at which the skeleton 202 followed is obtained, the computation of the intermediate skeleton 206 (and therefore also of the tracking loop 200) is not strictly necessary.
    [0155]
    [0156] Figure 3A illustrates an image of the simulated hand 310 and all the subsystems involved when interacting with various objects 222, and in particular with two rigid bodies: skeleton 202 followed by hand 300 followed by the user, skeleton 214 and tissue 216 from simulated hand 310, objects 218 touched in hand simulation 210 and objects 222 touched in virtual environment 220. Figure 3B shows another embodiment with the simulated hand 310 interacting with only one object 222, also a rigid body.
    [0157]
    [0158] The skeleton 202 followed by the hand 300 followed represents the configuration followed by the hardware and the hand follower software, while the skeleton 214 of the simulated hand 310 represents the articulated skeleton with joints 312 and bones 314. The joints 312 are modeled using energies Uj , which hold the endpoints of the bones 314 together and limit the angle of the joint. Both skeletons (202, 214) are connected using Ucrb energies , between each pair of bones through an intermediate layer (intermediate skeleton 206). In simulated hand 310, tissue 216 (represented as a mesh) is connected to skeleton 214 articulated through energies U st. The energy Ut represents the energy within tissue 216 due to deformations and contact. The simulated hand 310 is in contact with other objects 218. Between the fabric 216 and the objects 218 in contact, the friction energy Uf and the contact energy A are used to connect the fabric 216 with the objects 218 touched. Objects 218 in contact with simulation 210 of hand are connected to objects 222 in virtual environment 220 through the Ucrb energy between both objects.
    [0159]
    [0160] The description of the potential energies between the individual parts will now be explained in detail.
    [0161]
    [0162] Starting with the dynamics, due to the conservation of energy in a closed system, the sum of kinetic T energies and potential U energies is equal to some constant, the total energy in the system:
    [0163]
    [0164] T + U = constant (1)
    [0165]
    [0166] Alternatively, the Lagrangian L (q, q, t) of the system is defined as
    [0167]
    [0168] ¿(Q, q, t) = rCq.q.t) - l / (q, q, t), (2)
    [0169]
    [0170] with q the generalized coordinates and q the generalized speeds. Using the Euler-Lagrange equation
    [0171]
    [0172]
    [0173]
    [0174]
    [0175] the equations of the movement are obtained:
    [0176]
    [0177]
    [0178]
    [0179]
    [0180] giving the definition of the kinetic energy T = ^ qT Mq and assuming that the mass is constant over time, equation (4) expands to
    [0181]
    [0182]
    [0183]
    [0184] with M the generalized mass matrix and q the generalized accelerations. To obtain Euler's inverse time discretization, a direct difference approximation of the acceleration q is obtained using
    [0185]
    [0186]
    [0187]
    [0188]
    [0189] and the speed q generalized by:
    [0190]
    [0191]
    [0192] Substituting equation (7) into equation (6), the following approximation for general acceleration is obtained:
    [0193]
    [0194]
    [0195]
    [0196]
    [0197] That can be substituted in equation (5). This gives the following system of equations:
    [0198]
    [0199]
    [0200]
    [0201]
    [0202] From this relationship, the function h (x) can be defined,
    [0203]
    [0204]
    [0205]
    [0206]
    [0207] which corresponds to the optimality condition of an energy function E (q):
    [0208]
    [0209] E ( q) = ^ £ q TWq U, ( 11 )
    [0210]
    [0211] And approximate using equation (8) with q = qi + 1 the generalized positions. Therefore, the equations of motion are the energy gradients, that is, h (q) = dE (q) / dq. Minimizing the energy gradient for some configuration q, the energy is conserved, which can be formalized by
    [0212]
    [0213] q = argmin | h (q) |. ( 12 )
    [0214]
    [0215] There are several alternative approaches to minimize the equation (12). One approach is to perform a first-order Taylor expansion in equation (10) in order to find the unknown qi + 1 using h (ql + 1) = h (ql)
    [0216] Newton's method. In the special case where the derivative of h (q) is constant, both approaches achieve exactly the same procedure. The derivation using Newton's method is used herein. For a given / (x) function, roots can be found using the following iterative method:
    [0217] since the initial position of x is close to the real root. Finding the roots of equation (10) results in the following iterative procedure:
    [0218]
    [0219]
    [0220]
    [0221]
    [0222] With z the index of integration, which does not distinguish from stage i of time. The Hessiana becomes
    [0223]
    [0224]
    [0225]
    [0226]
    [0227] which is a constant matrix of the U quadratic function. As will be explained later, the scholastic potential energy U for all defined energies; therefore, an iteration of Newton's method minimizes equation (12). Now using qz = ql and qz + 1 = ql + 1, the following derivation is obtained:
    [0228]
    [0229]
    [0230]
    [0231]
    [0232] which can be recognized as a reverse Euler method. Since the potential U energy depends on the generalized coordinates, only the Jacobian or the Hessian of the potential energy is required with respect to q. Once qi + 1 is obtained, the new generalized positions are obtained by
    [0233]
    [0234] qI + 1 = q ‘Afq‘ + 1. (17)
    [0235]
    [0236] With regard to elastic tracking, to control a virtual hand following the movement of a real hand, the virtual connection between the virtual hand and the followed hand is applied. Applying the hand movement followed directly in the virtual hand seems logical, but it can cause several problems. If the movement is directly applied to the virtual hand, the amount of forces applied to the objects held in the virtual hand can grow quickly and can become unrealistic. Unrealistic behavior occurs because the actual movement of the hand is not limited when the virtual hand is obstructed by an object. Another problem originates from poor monitoring results. If the data obtained from the handheld follower is too loud or discontinuous, excessively large forces can be computed in the handheld model due to large reported speeds. To solve this, a virtual connection scheme is applied between simulated hand 310 and hand 300 followed. This virtual connection absorbs the excessive forces introduced by hand tracking. In addition, this connection absorbs the forces that could have been generated by the user's movement. For example, when the user grabs a virtual object using the virtual hand, or when the movement of the virtual hand is obstructed by other virtual (static) objects.
    [0237]
    [0238] A layered configuration is now followed. The movement of the user's hand is typically tracked at a different simulation speed than the virtual hand simulation. In order to make a stable connection of the handheld simulation 210 with the handheld tracking method, a separate simulation (tracking loop 200) is preferably performed at a higher simulation rate. The tracking loop 200 samples the follower data (hand 300 followed) and from the skeleton 214 fully articulated in the virtual hand through a synchronized copy 204. In this simulation, the "objective" intermediate skeleton 206 (an intermediate skeleton layer) is updated over time, also taking into account the connection between the entire skeleton and the intermediate skeleton. Therefore, this virtual connection system can Be considered as three layers of skeleton.
    [0239]
    [0240] 1. Configuration followed by hand skeleton, skeleton 202 followed, obtained by a hand follower.
    [0241]
    [0242] 2. Skeleton 206 intermediate.
    [0243]
    [0244] 3. Complete articulated skeleton 214 containing bones 314 and joints 312 in the hand.
    [0245] First, the hand follower establishes the position and orientations of the bones in the first layer, and can be connected to the second layer directly using the following energy function:
    [0246]
    [0247]
    [0248] with the positions of the bone at intermediate and the configuration of the bone b followed and A9eab its difference in its axis angle orientations. In addition, since the followed bone configuration is provided, only these energies are minimized for the intermediate bones, while the values of the followed consecutive configurations are maintained during the simulation stage. When this energy function is minimized, the configuration of the intermediate bones will follow the movement of the configuration followed.
    [0249]
    [0250] The connection between the intermediate layer (intermediate skeleton 206) and the complete articulated skeleton 214 is slightly different. Since the tracking and connection are performed using a much higher refresh rate than the full-hand simulation, each simulation simulates its own skeleton example. To connect these examples to the intermediate layer, equation (18) is used. First, a synchronized copy 204 of the complete skeleton 214 in the hand simulation 210 is connected to the intermediate skeleton 206 using equation (18) for which the positions and orientations of the skeleton in the hand configuration are kept constant. This allows changing the positions and orientations in the intermediate skeleton 206. Secondly, a similar connection is made between a synchronized copy 212 of the intermediate skeleton 206 and the complete skeleton 214 in the hand model. Now the positions and orientations of the intermediate skeleton remain constant, although the complete skeleton 214 (and therefore the complete hand) can be updated. Minimizing these energies allows updates of the intermediate skeleton 206 given the connection energies of the configuration followed and the handheld simulation 210. In addition, changes in energy between the intermediate layer and the skeleton of the hand result in updates of the virtual hand configuration in an asynchronous manner. Since the minimization of the energies between the skeleton 202 followed, the intermediate skeleton 206 and the synchronized copy 204 of the skeleton 214 of the hand operates at a higher refresh rate, stability is guaranteed. In addition, computed energies and corresponding forces can be used to control external haptic devices.
    [0251]
    [0252] The virtual hand model consists of different combined simulations for which different energy functions can be defined and added to the system described in equation (16). The hand model consists of an articulated body skeleton 214, a deformable skin (fabric 216), and a connection between the skin skeleton, and the contact between other objects 218 and the virtual hand.
    [0253]
    [0254] The bones 314 and the joints 312 form a skeleton 214 that is in a rigid articulated body simulation that forms the third layer in the virtual connection. The kinetic energy T for rigid bodies is computerized by
    [0255]
    [0256]
    [0257] with vc the center of mass velocity, 2 the angular velocity and I the moment of inertia. Joints 312 between rigid bodies define a potential energy Uj between the two rigid bodies. For a spherical joint this potential Uj energy can be described as follows:
    [0258]
    [0259] Uj = ^ k \ xca + ra - xcb - rb | 2, (20)
    [0260]
    [0261] which defines the potential between two endpoints of the bones 314 to form the joint 212. The positions xca and xcb describe the positions of the centers of mass of both bones a and b. The vectors ra and rb are the vectors from the centers of mass of the endpoints of the bones 314 that are used in the joint 312. Depending on the type of joint, additional energies can be defined to restrict the relative movement between the bodies in a particular axis
    [0262]
    [0263] Above this articulated body simulation (skeleton 214), a simulation of a finite element method is used to simulate skin deformation (tissue 216). Since the deformation of the skin is not linear, a tension limitation method / material is used to mimic the behavior of human skin. In the first stage of the deformation, the stiffness is relatively low, small forces result in a large deformation. As the deformation increases, the stiffness of the skin increases, resulting in minor deformation and increased tension in the skin. Because of this, the skin is capable of generating greater forces on the objects.
    [0264]
    [0265] The potential Ut energy for a deformable body can be described by
    [0266]
    [0267] Ut = jn W ( F ( q)) dq, (21)
    [0268]
    [0269] With 5 the potential energy density at a point within the material and F the deformation gradient. For a function 5 of quadratic potential energy density, a linear elasticity model can be easily obtained. By dividing the computational domain into a number of tetrahedron elements, 5 can be approximated by each element. In order to limit the voltage, 5 will be scaled once a density 5 max of threshold energy is reached. For example:
    [0270] W = Wa - kc [—q- ~ 1), (22) q 2 c Vwmax )
    [0271]
    [0272] with W9 the current energy density, Wmax the energy threshold and kc an additional penalty energy density. The energy density W9 can be originated from a linear consecutive elasticity model.
    [0273]
    [0274] The following finite element method (MEF) can be used. Given a particular energy density function W, the first voltage tensioner P (F) of Piola-kirchhoff can be obtained through the derivative of the function w.r.t. of energy density of the gradient F of the information, that is,
    [0275]
    [0276]
    [0277]
    [0278]
    [0279] The strain gradient F for a particular tetrahedron element refers to the initial Dm configuration with the current Ds configuration, that is Ds = FDm such that DjD ^ 1. Both Ds and Dm contain 4 column vectors with the current and initial coordinates of all the vertices of the elements, where component 4a contains a one ([30]).
    [0280]
    [0281] Similar for a Cauchy tension, a tension vector is obtained
    [0282]
    [0283] T = P (F) n, (24)
    [0284]
    [0285] with n the normal surface of the cross-sectional area. By discretizing the interior of an object using tetrahedron elements, the tension vectors for each surface element of each tetrahedron element can be computed. The force generated by the tension depends on the areas of these surface elements. By cross-sectional area, a corresponding force that contributes to each of the three nodes of the surface element can be computed. Therefore, the total force in a particular tetrahedron mode is the sum of the adjacent cross-section tensions, divided by three, that is,
    [0286]
    [0287]
    [0288]
    [0289]
    [0290] With Az nz the normal weighted area of an incident face at vertex i. For a tetrahedron element, node i is only shared by 3 faces of the same element. The derived force is computed as:
    [0291] d 2 Ut _ df_ _ ¿HdF _ / dP (F) ^ áF
    [0292] ( 26 )
    [0293] dq 2 dq dF dq V dF '/ dq'
    [0294]
    [0295] Since the force and the derivatives of the force depend on the Jacobian and the Hessian of the energy density function 5, they are derived for the case in which the energy density exceeds the threshold. The Jacobian and the Hessian of this modified energy density function in equation (22) becomes:
    [0296]
    [0297]
    [0298]
    [0299]
    [0300] Due to the increased energy density, the tension and therefore formation will be reduced.
    [0301] The deformation gradient F contains the rotation, deformation and possible reflections (reversals) of each element, where the energy density function 5 should be invariable to the rotation for isotropic materials. To separate the rotation from the deformation, a Polar Decomposition or a Singular Value Decomposition (SVD) can be used. A popular approach is the co-rotating MEF method (18). The co-rotating MEF computes the tension in a frame without turning by rotating in the movement in the frame without turning. Given the displacement without rotation the tension and forces are computed in the frame without rotation. Finally, the result is turned back to the rotated frame of the displacement. The co-rotating method can be summarized by:
    [0302]
    [0303] f = —ReKR £ q ReKq0, (29)
    [0304]
    [0305] With K the stiffness matrix of the element and Re the rotation of the element obtained through a polar decomposition. Since q0 is already in the frame without rotation, no correction is applied for qC. In the case of a linear elasticity, K remains constant throughout the entire simulation.
    [0306]
    [0307] Alternatively, since those derived from the energy density function are not constant for the stress limitation approach, the description of the forces is followed as described in the previous paragraph. Using the SVD, the deformation gradient can be decomposed through F = uFvT, with F a diagonal matrix containing the singular values of the deformation gradient and U and VT two rotation matrices. Since the density function 5 of energy is invariable to spin for isotropic materials, the first voltage of Piola-Kirchhoff can be computed by
    [0308]
    [0309] TO
    [0310] P (F) = UP (F) Vr = U ^ ^ V 7-. (30)
    [0311] dF
    [0312]
    [0313] Since F is a diagonal matrix that contains the G1, G2, G3 deformations in the three main axes, the computation of Wq and its Jacobiana and Hessiana become simple. When the determinant of the deformation gradient is negative, the element is inverted.
    [0314]
    [0315] In order to handle this investment, the U and VT turn matrices and the F matrix need to be corrected in such a way that the turn matrices are free of any reflection, see (9). When large deformations are expected, also those derived from the turns need to be included, which requires those derived from the SVD, see (23) and (29).
    [0316]
    [0317] Regarding the skeleton-tissue connection, the MEF simulation is connected to the underlying articulated body simulation defining an energy function between the tissue 216 and the rigid bodies in the skeleton 214. Each node 316 belonging to an element in the tetrahedrization is intercepted by a rigid bone 214, is fixed to that bone 214 using the following energy:
    [0318]
    [0319]
    [0320]
    [0321]
    [0322] with xc the center of mass of the rigid body, Rb the body orientation and rcn the vector from the center of mass to the interpolated position xB =
    [0323]
    [0324] The contact between the virtual hand and another object is modeled using a penalty energy between nodes in the virtual hand and an anchor node that is arranged on the surface of the body. When a particular node of the skin tissue 216, a vertex xn, penetrates the surface of an object 218, its distance f (x) indicated with reference to the body surface becomes negative. When this occurs, there is a configuration of both object 218 and vertex xn between the current and previous time stage, where xn is arranged exactly on the surface of the body. In this location, an anchor node x0 is initialized and the normal contact n is sampled, as shown in Figure 4A. Given the anchor node x0 and the vertex xn, You can define a potential energy. Here we distinguish between normal tangential energy. When a skin node is in contact with the body, energy A potential in the normal direction becomes:
    [0325]
    [0326]
    [0327]
    [0328]
    [0329] With x0 the position of the anchor node on the surface of the object 218, n the normal contact, kc the contact stiffness and w a weighting value.
    [0330]
    [0331] Similarly, a bi-normal and tangential energy can be defined by replacing the normal ones with the tangent vector t 1 or the bi-normal vector t 2. The vectors t 1 and t 2 are both perpendicular to each other. In addition, t 1 points to the anchor node n0 in the direction of xn.
    [0332]
    [0333] Figure 4A represents the contact model used for the interaction between the tissues 216 and the object 218. A vertex xn of the skin penetrates the object (distance f (x) <0) in k n. Then, the insertion of the trajectory of the vertex xn with the surface 402 of the object 218 is computed. At the point of intersection, the anchor node x0 is initialized and the normal n =
    [0334]
    [0335] The energy model just described minimizes the distance between the anchor x0 and the contact node when a static friction is applied, that is, when both surfaces do not slide together. To determine whether static or kinetic friction should be applied, Coulomb's friction model is approximate. When a force ft in the tangential direction is larger than the force fn in the normal direction multiplied by the coefficient w of nrr kinetic friction should be applied, that is, | ft | > wl | fn II, with fn = - n / aq. In this case, the tangential force is sustained by the normal force multiplied by w so that the kinetic friction is applied, that is, the surfaces are allowed to slide together, as shown in Figure 4B. If a kinetic friction is applied, the anchor x0 is allowed to slide over the surface 402. Given the normal force, the anchor x0 is moved towards xn such as the corresponding friction force relative to the Coulomb friction model.
    [0336]
    [0337] To sustain the energy and force in the tangential direction, first the magnitude of the friction force is held in accordance with the Coulomb model for friction. Then, the anchor point should move on the surface 402 of the object 218 in the direction of the sliding speed. The displacement should be chosen in such a way that the magnitude of the frictional force in the next stage of time approximately satisfies Coulomb's law, that is | ft | wl | fn II- The potential Uf energy tangential in case of kinetic friction then becomes:
    [0338]
    [0339] Uf = ^ k cw n | ttr (x c ra - x n) | 2. (33)
    [0340]
    [0341] By updating the anchor position, a portion of the potential energy of the system is removed - This energy is converted into thermal energy, which is not modeled in this simulation.
    [0342] In order to determine whether static friction should be applied, the magnitude of the tangential friction force should be compared with the normal contact force multiplied by the coefficient of friction. Yes | ft | <wl | fn |, static friction should be applied and the anchor is not allowed to slide over the surface.
    [0343]
    [0344] Objects in the virtual world can interact with each other and with the virtual environment. These objects are usually simulated using a physical engine (virtual environment 220) that takes care of these interactions and applies the laws of motion to these objects. In order to consider the interaction between the simulated hand 310 of the objects, a connection to the virtual world must be applied, similar to the virtual connection used between the hand follower and the simulated hand. In this case, each object exists in both simulations, one in the virtual environment 220 and one that is simulated together with the virtual hand in the hand simulation 210. If the object is rigid, the coupling energy is U'crb, described in equation (18), but takes the object of both simulators (object a of hand simulation 210 and object b of virtual environment 220). If the object is flexible, the connection energy is proportional to the Euclidean norm of the difference between the states of objects a and b. The connection energy minimizes the difference between both cases of the objects (218, 222). This energy term is applied to each object in both simulators. When these energies are minimized, a pair of status representations remains constant since they are not computed by the same simulator (using copies 224 and 226 synchronized in Figure 2).
    [0345]
    [0346] In order to deliver a stable simulation of the complete hand model, individual aspects of the simulation are solved together using a large system of equations. This includes the bones 314 and the joints 312 in the articulated skeleton 214, the MEF simulation of the skin 216, the additional bodies, the connection between which the skin 216 and the bones 314, the connection of the objects and their representation in the virtual world, and the contact between skin and objects. To obtain this global system, the Jacobian and the Hessian of the potential energies with respect to q need to be computed for each defined energy. Here it contains the positions of the skin nodes and the state of the bodies. In addition, derivatives of kinetic energies must be computed and stored in the same system, see equation (4). This large system is solved for generalized rates of unknown that are applied later, see equation (17). Said large system allows a staggered implicit time scheme to be carried out correctly, which increases the accuracy and stability of the simulation. Figure 2 shows how the terms of individual energy are connected.
    [0347]
    [0348] As mentioned earlier, an important advantage of the present method is that it allows uninterrupted integration with the physical motors ready for use. Building on this feature, the present hand simulation method has been encapsulated in a library that simplifies integration with other modules. In particular, the library constitutes an asset of a game engine, and the soft-hand model can be added to a virtual environment through a simple drag-and-drop operation. Figure 5 shows a brief overview of the simulation library 500 with different modules.
    [0349]
    [0350] The simulation library 500 is a modular library for handheld simulations that allows easy integration with third-party tools 510 (hardware and / or software) and in particular with different types of tracking devices 512. In addition, the library 500 can be used to control various types of 512 haptic devices and 516 touch devices through specific controllers or modules. Library 500 also provides an interface for third-party simulation environments 518, such as Unity or Unreal Engine, to share information regarding the state of physical objects that are allowed to interact with the present hand simulation. In addition, these virtual 518 environments are used to visualize the state of the simulated hand. Due to the modular configuration, custom controllers can be developed for certain hardware, also allowing easy integration with existing software.
    [0351]
    [0352] The main module of library 500 is invoked by physical / third-party game engines. It allows the user to load specific drivers for 514 and 516 touch devices and for various types of 512 motion tracking devices. The user is also able to create modules to customize devices using an SDK (software development kit). The main module contains the simulation of the hand and is performed only by the main module. It does not use the host physics engine. For each type of physical / game engine supported, the host objects / classes are intended to be used by the user to interact with the library. These host objects are responsible for invoking the library, providing it with additional information regarding the simulated environment on the host physical engine. The host object is also responsible for displaying the status of the simulation performed in the simulation library and providing the virtual environment with additional responses that interact with the objects defined in the host physical engine.
    [0353] Tracking modules are used to generalize various hand / movement tracking methods, for example, Kinect, LeapMotion, OptiTrack or other methods. Each module is responsible for acquiring the positions and orientations of the fingers and the position of the hand, transforming the data into the appropriate format and feeding the simulation of the hand with appropriate movements.
    [0354]
    [0355] Haptic representation modules are used to control 514 haptic devices. API 520 provides the controller with information on the position followed and the orientation of the bones, and the current position and presentation of the simulated bones. In addition, all forces and pairs computed are provided by the connection between the bones followed and simulated. Given this information and some device-specific information, the controller can compute response forces or actuator positions in order to provide the end user with realistic haptic feedback through the device.
    [0356]
    [0357] The touch representation modules provide an interface to the touch representation devices 516. Touch devices provide the end user with tactile feedback of the objects touched in the virtual world. For each finger in the simulation, tactile information can be provided by the collision events generated. Based on this information, the module can compute a response or movement force for the device.
    [0358] In order to control the various types of modules, each with a different set of parameters, the user can send messages through the main interface. Library 500 distributes the message to all loaded modules so that they can be configured and / or controlled directly. For example, for specific types of haptic devices, a few engines must first be enabled in order to be functional.
    [0359]
    [0360] Figure 6 shows an embodiment of the hardware configuration used to run a simulation and interact with virtual objects in a virtual environment shown through a head-mounted display 610 (HMD). The simulation takes as input a hand configuration, a hand 300 followed, obtained by a hand follower or a hand tracking device 620, such as a Leap Motion controller, through some hand tracking method. The hand-tracking device 620 may be connected to the head-mounted screen 610 in order to increase the user's workspace 630.
    [0361] Depending on the tracking method, a wide variety of cameras can be used to track the movement of the user's hand. For example, stereo depth cameras or standard depth cameras can be used. The hand tracking method detects the position and shape of the user's hand and the present hand simulation method acquires the hand configuration by obtaining this information from the hand tracking device 620 (using the 512 follower device controller Figure 5). For example, if a Leap Motion controller is used as a hand tracking device 620, the user's hand configuration is captured using a stereo depth camera and reconstructing the bone configuration of the captured images.
    [0362]
    [0363] The complete simulation performed on a data processing device, such as a 650 computer or a laptop that uses a multi-core CPU. The visualization of the virtual environment is done using a game engine ready to use and the GPU to visualize the hand simulation in the HMD 610 and perform the simulation of the objects in the game engine. Hand simulation can be implemented in a connector created for a game engine (for example, Unreal Engine 4, Unity). Each game engine is able to interact with hand simulation through the interface provided by the connector. In addition, the game engine can provide the hand simulation with additional information regarding the objects that are created by the game engine.
    [0364]
    [0365] The visualization of the handheld simulation can be performed on a display device, such as a monitor, a television or a 610 device mounted on the head. Fire engines can support the use of a head-mounted display to display the virtual world in an immersive way to the user. The head mounted display 610 may be equipped with additional sensors to obtain the orientation, acceleration and position of the head mounted display. A head tracking device 660 can be used to capture the position of the head-mounted screen 610 (such as headphones with Oculus Rift or HTC Vine headphones).
    [0366]
    [0367] The hand tracking device 620 may be connected to the head mounted display 610 and the information of these sensors transferred to the handheld simulation. The sensors of the handheld tracking device 620 can only report positions and orientations with respect to the camera. Considering also the data of the sensors of the head-mounted screen, the absolute position of the hand 300 followed and its configuration can be reconstructed, significantly increasing the user's workspace. In the default configuration, the 620 Leap Motion controller camera is located in front of the user on a flat surface, but the position and orientation of the camera can be anywhere in the virtual and real world.
    [0368]
    [0369] Since the library computes the forces in the elastic connection and the contact forces between the hand and the object touched in the virtual environment, the library can compute feedback forces for different types of haptic and tactile devices, such as a 670 haptic device handheld (for example, a glove). For the experiments shown in [17], a hybrid haptic and tracking device controller was created to compute a response force for the exoskeleton used, and uses the sensors in the exoskeleton to acquire the finger configuration. In addition, touch device drivers are implemented to control the devices described in [24] using the methods described in this document.
    [0370]
    [0371] The described hand simulation and the connection method allow the user to use their hands to interact with the virtual world in a physically correct way. The forces that are computed when the virtual hand is in contact with the virtual objects or the forces computed in the virtual connection, can be used to control external devices for haptic and tactile feedback. Thanks to the connection method and the way in which the virtual hand simulation is configured, a very robust interaction mode is obtained that prevents the user from interacting with the virtual world in a non-physical way. Using a nonlinear deformation model, virtual skin behavior mimics the behavior of real tissue.
    [0372]
    [0373] In the configuration shown in Figure 6, the user carries an HMD in which the hand follower is fixed. The head keeps track of the position of the HMD and the hand follower. The sensor, the HMD and the hand follower are connected to an end user's computer (CPU + GPU). When the simulation is running, the HMD shows the virtual environment to the user and the virtual world screen responds to the user's movement. When the hand follower detects the user's hand, the hand position with respect to the hand follower is obtained. When this position and the position and orientation of the HMD, the simulated hand is shown (to the user) with respect to the HMD. When the user of a virtual object through HMD, the user is able to move his hand closer to the virtual object. When the virtual hand is close enough to the virtual object, the user can interact with the virtual object using his own hand. Alternatively, the hand tracker can be configured in a fixed position close to the user (for example, at a table in front of the user).
    权利要求:
    Claims (16)
    [1]
    1. A method implemented by computer to simulate the behavior of a hand that interacts with objects in a virtual environment, the method comprising:
    repeatedly acquire a hand configuration of one hand (300) followed from data captured by a hand tracking device (620), the hand configuration that includes information to establish the position and orientation of bones that form a skeleton (202) followed by the hand (300) followed;
    repeatedly computing a simulated hand-held configuration (310) in a hand-held simulation (210) that interacts with at least one object (222) in a virtual environment (220), the simulated hand (310) comprising a deformable nonlinear fabric (216) and a skeleton (214) articulated with joints (312) and bones (314);
    characterized in that the method further comprises:
    establish a connection between the skeleton (214) of the simulated hand (310) and the skeleton (202) followed by using energies ( U crb) of connection between pairs of corresponding bones considering their relative positions and orientations,
    determine an internal energy of the simulated hand (310), the internal energy comprising a potential energy ( Uj ) associated with the joints (312) of the skeleton (214), a potential energy (U) associated with the tissue (216), and an energy function ( U st) associated with the connection between the tissue (216) and the skeleton (214);
    establish a friction contact between the simulated hand (310) and at least one object (218) in the hand simulation (210) using potential energies in the normal ( U n) and tangential ( Uf ) directions between a node (xn) of the fabric (216) of the simulated hand (310) and a surface anchoring node (xa) (402), of the at least one object (218) of the hand simulation (210); Y
    Repeatedly compute an updated skeleton (214) and a simulated hand tissue (216) (310) minimizing the total energy ( Ucrb, Uj, Ut, Ust, Un, Uf ) involved.
    [2]
    2. The computer-implemented method of claim 1, wherein the step of establishing a connection between the skeleton (214) of the simulated hand (310) and the skeleton (202) followed comprises:
    establish a connection between a synchronized copy (204) of the skeleton (214) of the simulated hand (310) and an intermediate skeleton (206) using connection energies ( Ucrb) between each pair of corresponding bones;
    establish a connection between the skeleton (202) followed and the intermediate skeleton (206) using connection energies ( Ucrb) between each pair of corresponding bones;
    establish a connection between a synchronized copy (212) of the intermediate skeleton (206) and the skeleton (214) of the hand (310) using connection energies ( U crb) between each pair of corresponding bones;
    Repeatedly compute an updated intermediate skeleton (206) minimizing the connection energies (Ucrb) involved.
    [3]
    3. The computer-implemented method of claim 2, wherein the computation of an updated intermediate skeleton (206) is performed at a higher rate than the computation of the updated skeleton (214) of the simulated hand (310).
    [4]
    4. The method implemented by computer according to any of the preceding claims, wherein the step of computing a simulated hand-held configuration (310) comprises modeling the joints (312) using potential energies ( Uj ) to maintain the points Bone ends (314) that form the joint (312) together and limit the angle of the joint (312).
    [5]
    5. The method implemented by computer according to any of the preceding claims, wherein the step of computing a simulated hand configuration (310) comprises computing the tissue deformation (216) using a finite element method, in where the deformable tissue (216) is described considering the potential energy ( U t) having a function W of quadratic potential energy density, and where the computational domain is divided into a set of tetrahedron elements.
    [6]
    6. The method implemented by computer according to any of the preceding claims, wherein the step of computing a simulated hand configuration (310) comprises establishing a connection between the skeleton (214) and the tissue (216) of the hand (310) simulated using an energy function ( U st) between bones (314) in the skeleton (214) and nodes (316) of the tissue (216) corresponding to the intersection of bones (314) and tissue (214) .
    [7]
    7. The method implemented by computer according to any of the preceding claims, wherein the at least one object (222) is a rigid body, the method further comprising:
    establish a connection between the at least one object (218) that interacts with the hand (310) simulated in the hand simulation (210) and the corresponding object (222) simulated in the virtual environment (220) using energies ( U'crb ) of connection considering the difference in position and orientation between pairs of objects (218, 222) and
    repeatedly compute at least one object (218) updated in the handheld simulation (210) minimizing the connection energies ( U'crb ).
    [8]
    8. The method implemented by computer according to any of claims 1 to 6, wherein the at least one object (222) is a flexible body, the method comprising:
    establish a connection between the at least one object (218) that interacts with the hand (310) simulated in the hand simulation (210) and the corresponding object (222) simulated in the virtual environment (220) using energies that consider the difference in the state between pairs of objects (218, 222) and
    Repeatedly compute at least one object (218) updated in the handheld simulation (210) minimizing the connection energies.
    [9]
    9. The method implemented by computer according to any of the preceding claims, which comprises representing on a display device (610) the simulated hand (310) interacting with the at least one object (222) in the virtual environment (220) .
    [10]
    10. A system for simulating the behavior of a hand that interacts with objects in a virtual environment, comprising a data processing device (650) configured to:
    repeatedly acquire a hand configuration of one hand (300) followed from data captured by a hand tracking device (620), the hand configuration that includes information to establish the position and orientation of bones that form a skeleton (202) followed by the hand (300) followed;
    repeatedly computing a simulated hand-held configuration (310) in a hand-held simulation (210) that interacts with at least one object (222) in a virtual environment (220), the simulated hand (310) comprising a deformable nonlinear fabric (216) and a skeleton (214) articulated with joints (312) and bones (314);
    characterized in that the data processing device (650) is further configured to:
    establish a connection between the skeleton (214) of the simulated hand (310) and the skeleton (202) followed by using energies ( U crb) of connection between pairs of corresponding bones considering their relative positions and orientations,
    determine an internal energy of the simulated hand (310), the internal energy comprising a potential energy ( Uj ) associated with the joints (312) of the skeleton (214), a potential energy (U) associated with the tissue (216), and an energy function ( U st) associated with the connection between the tissue (216) and the skeleton (214);
    establish a friction contact between the simulated hand (310) and at least one object (218) in the hand simulation (210) using potential energies in the normal ( U n) and tangential ( Uf ) directions between a node (xn) of the fabric (216) of the simulated hand (310) and a surface anchoring node (xa) (402), of the at least one object (218) of the hand simulation (210); Y
    Repeatedly compute an updated skeleton (214) and a simulated hand tissue (216) (310) minimizing the total energy ( Ucrb, Uj, Ut, Ust, Un, Uf ) involved.
    [11]
    11. The system of claim 10, comprising a hand tracking device (620) for repeatedly acquiring the hand configuration (300) followed by a user (630).
    [12]
    12. The system according to any of claims 10 to 11, comprising a display device configured to represent the simulated hand (102) that interacts with it at least one object (222) in the virtual environment (220).
    [13]
    13. The system of claim 11, wherein the display device is a monitor (610) mounted on the head.
    [14]
    14. The system according to any of claims 10 to 13, wherein the data processing device (650) is configured to compute forces at simulated hand contact points (310) generated by interaction with objects (222 ) in the virtual environment (220), the system which further comprises a handheld portable haptic device (670) configured to apply a haptic feedback response in the hand (300) followed by a user (630) according to the computed forces .
    [15]
    15. A computer program product to simulate the behavior of a hand that interacts with objects in a virtual environment, characterized in that it comprises a program code usable by computer to, when executed in a processor, perform the steps of the method implemented by computer defined in any one of claims 1 to 9.
    [16]
    16. A computer readable storage medium comprising instructions for causing a processor to carry out the steps of the computer implemented method defined in any one of claims 1 to 9
    类似技术:
    公开号 | 公开日 | 专利标题
    US10026233B2|2018-07-17|Efficient orientation estimation system using magnetic, angular rate, and gravity sensors
    Borst et al.2005|Realistic virtual grasping
    Weinstein et al.2006|Dynamic simulation of articulated rigid bodies with contact and collision
    Aristidou et al.2013|Real-time marker prediction and CoR estimation in optical motion capture
    Otaduy et al.2006|A modular haptic rendering algorithm for stable and transparent 6-dof manipulation
    JP4463120B2|2010-05-12|Imitation robot system and its imitation control method
    JP2021529380A|2021-10-28|Methods and systems for interpolating heterogeneous inputs
    Saupin et al.2008|Contact model for haptic medical simulations
    Kenwright2012|Inverse kinematics–cyclic coordinate descent |
    US20170185141A1|2017-06-29|Hand tracking for interaction feedback
    Talvas et al.2015|Aggregate constraints for virtual manipulation with soft fingers
    Jin et al.2000|General constrained deformations based on generalized metaballs
    US10013792B2|2018-07-03|Methods and systems for computer-based skin animation
    Verschoor et al.2018|Soft hand simulation for smooth and robust natural interaction
    Pihuit et al.2008|Hands on virtual clay
    JP2021530815A|2021-11-11|Posture space dimension reduction for virtual character posture space deformation
    Teleb et al.2012|Data glove integration with 3d virtual environments
    JP2017049238A|2017-03-09|Method and system for vision measure for digital human models
    ES2725128B2|2020-07-22|Method implemented by computer, system and product of the computer program to simulate the behavior of a hand that interacts with objects in a virtual environment.
    Cui et al.2008|Simulation of grasping deformable objects with a virtual human hand
    Ji et al.2017|Motion trajectory of human arms based on the dual quaternion with motion tracker
    CN110008873B|2021-06-22|Facial expression capturing method, system and equipment
    KR102239957B1|2021-04-13|Virtual joint sphere based motion modeling method and device
    Yang et al.2011|Real-time deformations simulation of soft tissue by combining mass-spring model with pressure based method
    Nasim et al.2018|Physics‐based assistive grasping for robust object manipulation in virtual reality
    同族专利:
    公开号 | 公开日
    WO2019175460A1|2019-09-19|
    US20200393899A1|2020-12-17|
    ES2725128B2|2020-07-22|
    EP3751433A4|2021-03-31|
    EP3751433A1|2020-12-16|
    US11093037B2|2021-08-17|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    US20180024635A1|2016-07-25|2018-01-25|Patrick Kaifosh|Methods and apparatus for predicting musculo-skeletal position information using wearable autonomous sensors|
    US8576253B2|2010-04-27|2013-11-05|Microsoft Corporation|Grasp simulation of a virtual object|
    US8994718B2|2010-12-21|2015-03-31|Microsoft Technology Licensing, Llc|Skeletal control of three-dimensional virtual world|
    JP2014071499A|2012-09-27|2014-04-21|Kyocera Corp|Display device and control method|
    US9552673B2|2012-10-17|2017-01-24|Microsoft Technology Licensing, Llc|Grasping virtual objects in augmented reality|
    US10318008B2|2015-12-15|2019-06-11|Purdue Research Foundation|Method and system for hand pose detection|
    KR20190110539A|2016-12-13|2019-09-30|딥모션, 인크.|Enhanced Virtual Reality System Using Multiple Force Arrays for Solver|
    WO2018187171A1|2017-04-04|2018-10-11|Usens, Inc.|Methods and systems for hand tracking|
    法律状态:
    2019-09-19| BA2A| Patent application published|Ref document number: 2725128 Country of ref document: ES Kind code of ref document: A1 Effective date: 20190919 |
    2020-07-22| FG2A| Definitive protection|Ref document number: 2725128 Country of ref document: ES Kind code of ref document: B2 Effective date: 20200722 |
    优先权:
    申请号 | 申请日 | 专利标题
    ES201830260A|ES2725128B2|2018-03-15|2018-03-15|Method implemented by computer, system and product of the computer program to simulate the behavior of a hand that interacts with objects in a virtual environment.|ES201830260A| ES2725128B2|2018-03-15|2018-03-15|Method implemented by computer, system and product of the computer program to simulate the behavior of a hand that interacts with objects in a virtual environment.|
    US16/971,521| US11093037B2|2018-03-15|2019-03-13|Computer-implemented method, system and computer program product for simulating the behaviour of a hand that interacts with objects in a virtual environment|
    PCT/ES2019/070170| WO2019175460A1|2018-03-15|2019-03-13|Computer-implemented method, system and computer program product for simulating the behaviour of a hand that interacts with objects in a virtual environment|
    EP19768022.6A| EP3751433A4|2018-03-15|2019-03-13|Computer-implemented method, system and computer program product for simulating the behaviour of a hand that interacts with objects in a virtual environment|
    [返回顶部]